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Abstract — Augmented reality is an advanced 
computational visualization technology that alters how 
users in the real world can perceive the virtual 
information. The use of this technology for EAC/FM is 
being widely investigated. In the scope of dam safety, the 
constant analysis of concrete behavior is mandatory, 
searching for clues of pathologies such as cracks. Cracks 
are relatively common in concrete structures, 
nevertheless they need to be surveilled due to the risks 
they offer. The surveillance of cracks involves exhaustive 
tasks, and for dams, it consists in the execution of a set of 
complex tasks that demands access to accumulated data 
and information. Augmented reality can contribute with 
the visualization process of this information, diminishing 
the mental workload demand. This paper defines a 
hierarchical taxonomy of the tasks that are needed in this 
domain, using Berliner's taxonomy to classify the tasks, 
enhancing the understanding of the points where the 
augmented reality can be used with better results. 
Keywords — augmented reality, task-analysis process, 
crack surveillance, dam crack, taxonomy. 

I. INTRODUCTION 

Augmented reality is an advanced interaction and 
visualization technology that has been widely used in a 
variety of domains, including entertainment, tourism, and 
help with task execution. This technology improves your 
perception of the environment around. In EAC/FM, 
augmented reality has been widely used in all phases of 
the enterprise life cycle. It can help by presenting, in a 
contextualized way, project information, schedule, 
planning, historical data, allowing the user to keep his 
attention on the task he is doing, avoiding context change. 
Computer systems that use augmented reality technology 
offer advanced means of interaction and visualization. 
They seek to expand the human perceptual system by 
combining virtual information in the real environment so 


that it contributes to inproving the conditions for 
accomplishing the tasks. Systems development is guided 
by an HCI project, resulting from a user-centered 
analysis. A good augmented reality system needs to know 
the environment where the tasks will be performed, the 
tasks and the users who will perform the tasks. 

For the success of the employment of augmented reality 
in AEC/FM it is important to identify the tasks and 
understand the information, perceptual and cognitive 
process required. The proposed taxonomy seeks to aid 
this. 

Employing augmented reality in the context of crack 
surveillance can help technicians to better perform the 
tasks required in the context. This article details the scope 
of the tasks, identifies and classifies them using Berliner 
taxonomy in order to map the perceptual, motor and 
cognitive components involved. 

II. AUGMENTED REALITY IN AEC/FM 

The use of augmented reality in the AEC/FM is being 
investigated approximately two decades ago. Much of the 
scientific work describes the development and use of 
prototypes and evaluates users' satisfaction with the 
experiences provided [1-14]. Some augmented reality 
prototypes focus on aiding in the monitoring or inspection 
of environments during construction [1,2,4—6,9]. Others 
for assembling aid [10,15-17], maintenance [18], for the 
execution of tasks that present uncertainties [19], for 
training [8,14,20], for remote execution of tasks [21]. 
Some scientific works have investigated the contributions 
of augmented reality in the management of a building 
[3,4,6], others assert that augmented reality may favor 
communication [6,10,22] and the safety of workers in the 
construction environment [6]. Several of these works 
already mentioned use the augmented reality in 
conjunction with the BIM - Building Information Model. 
There are also papers highlighting the benefits gained 
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from the use of augmented reality to explore the 
structured information of BIM [11], there are those who 
point out that it allows the information about the building, 
designed in the office, to be accessed at the construction 
site, in a contextualized way. 

Research indicates that the adoption of augmented reality 
in the area is an extended version of BIM because the 
technology addresses one of the key issues currently 
investigated in BIM: effective and efficient means to 
exploit highly integrated and organized information 
[11,23]. 

This set of scientific papers proves that the technology 
has a positive inpact on the practices of the sector and 
that can help in solving existing difficulties in many 
ways. Among the studies analyzed there were 
descriptions of the use of augmented reality in the 
infrastructure sector. The system proposed by Fujiwara: 
ARIiNER, uses the technology to assist the construction 
of containment dams, in hazardous access places (active 
volcano slopes), through the remote operation of 
machines [21]. Hammad, Garrett e Karimi defined a class 
of augmented reality applications called MARSIFT - 
Mobile Augmented Reality System for Infrastructure Field 
Tasks [24]. This paper presents, in a descriptive way, 
utilities of mobile augmented reality in the 
accomplishment of the necessary activities in the 
infrastructure constructions. The use of augmented reality 
to guide the movement of professionals to specific 
locations, to present evolution information related to 
monitoring or inspection activities, to communication 
between teams and to identify specific positions of 
interest through accurate tracking techniques is cited. The 
work of Zhou, Luo and Yang describes a case study for 
quality inspection in tunnels [25]. A model is retrieved 
and used to establish a virtual line that overlaps in the real 
environment allowing the assessment of structural safety 
through the measurement of differences between the 
model and the real environment view. 

The use of augmented reality in conjunction with BIM 
seeks mainly to [26]: 

• Reduce discrepancies between planned and executed 
solutions. 

• Reduce inefficiency in communication among 

professionals involved in activities. 

• Improve the perception and cognition of the 
professionals involved in decision making. 

• Facilitate access to information related to activities. 

• Improve the concentration and attention of the 
professional in the execution of the activities. This set of 
work shows that augmented reality is a technology with 
great potential in AEC/FM. It has proved adequate to 
enable access and sharing of information by the team of 
professionals who cooperate with each other. However, 


there are few studies that explore the use of augmented 
reality in infrastructure construction. 

III. CRACKS MONITORING 
The safety of a dam is maintained when its structural and 
operational integrity is preserved so that it satisfies 
behavioral requirements that seek to avoid failures in 
operation, dam and reservoir [27,28]. 

In the operation phase of a damit is essential to carry out 
a periodic surveillance and maintenance program [29]. A 
hierarchical classification of dam safety processes can be 
seen in the Fig. 1. 



Fig. 1: Hierarchical classification of dam safety 
processes 


Concrete behavior analysis occurs in the surveillance 
program; it investigates, among other things, the 
relationship of cracks with other dam events [30]. The 
presence of cracks in the structure is perceived during 
visual inspections (special, periodic or routine); however, 
in addition to identifying them, it is necessary to map 
them in order to understand the causative mechanism and 
the risks they pose to safety. Crack mapping consists of a 
thorough and exhaustive observation of the surfaces, in 
order to visually identify, record the shape and the 
relative location of cracks, according to some convention 
established by the organization. In addition, it is 
important to photograph and characterize cracks, building 
a historical record of its evolution over time. The records 
generate conditions to determine the cause and age of 
cracks [30].At each new mapping, a new set of individual 
data is collected. Although crack detection can occur in 
any type of inspection, it usually occurs only in special 
ones, where mapping is recommended. When a crack is 
identified, it has occurred some time ago. So it is not a 
simple task to make notes and estimates of the cause and 
age of the crack, but it is important to guide decision 
making. The data collected in the mappings need to be 
maintained to allow monitoring of evolution over time. 
The result of the mappings consists of internal and global 
reports, required by regulatory authority in the sector. 
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They present the current condition and evaluate the 
behavior of cracks by comparison with previous results, 
which support recommendations and decisions regarding 
maintenance and installation of instruments to monitor 
the behavior of active cracks [30]. In addition, the set of 
information resulting from the mapping is necessary for 
conclusions about the security offered by the structure. 
Although mapping inputs and outputs are well defined, 
there are several ways to do this, and their establishment 
in each organization considers, among other things, the 
size of the dam and the level of technology used in the 
context. The technologies employed directly interfere in 
the efficiency and in the effectiveness of the mapping. 

IV. HIERARQUICAL TAXONOMY 

Taxonomy is a term derived from the Greek (taxis - order 
and nomy - law, norm, rule) introduced by Candolle, in 
1813 [31,32]. Initially, they were used to classify living 
beings in a logical and scientific way; but is currently a 
method used in varied contexts as a resource to organize 
and classify conceptual units [31]. 

In the context of information processing models, Berliner 
et al. [33], have proposed a taxonomy that determines 
elementary units of behavior in relation to perception, 
mediation , motor and communication processes. 

The elementary units of behavior are declared by means 
of 32 verbs classified according to the type of activity 
performed. Although this taxonomy has been proposed 
more than 50 years ago, it is still used to determine 
human behavior and assist in describing operational 
procedures [34]. 

According to Eurocontrol [35] the classification proposed 
by Berliner et al. is useful to explain the behaviors that 
are common in tasks that require some type of interaction. 

The elementary units of behavior classified according to 
the processes are shown in Fig. 2. 
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Fig. 2.‘Berliner's taxomony [34] 

In the AEC/FM, taxonomies are used to classify the 
knowledge about the work required in the context so as to 
provide the means for the development of specific 
analyzes. Everett proposed a hierarchical taxonomy of 
construction-related tasks to analyze tasks and point out 
the most suitable ones for automation [36]. Everett’s 
taxonomy is composed of nine levels : Industry, Sector, 
Project, Division, Activity, Basic Task, Motion, 
Orthopedics and Cell. 

In order to analyze the tasks identified in the construction 
context, Everett took into consideration that humans are 
better able to perform tasks with predominance of 
improvised actions, or that have uncertain information 
and that require judgments based on experience and 
perception of complex stimuli. While machines are 
suitable for storing and retrieving information and 
performing repetitive jobs in a short time, without being 
distracted by external factors. 

Dunston and Wang proposed another hierarchical 
taxonomy for the context of the AEC/FM [37]. In this 
case the operations and tasks were categorized. The main 
objective was to provide the conditions to analyze them 
on issues related to the use of mixed reality. According to 
the authors, the taxonomy provides the following 
benefits: 

• It facilitates identifying the opportunities to explore 
the mixed reality in the AEC/FM. 

• Establishes methodologies for mapping technology to 
context tasks. 

• Identifies the core tasks of the AEC/FM. 

This taxonomy proposed by Dunston and Wang is 
composed of five categories: Application Domain, 
Application specific operation, Operation specific 
activity, Composite tasks and Primitive tasks, as shown in 
Fig. 3. 



Low General 

Tasks complexity Tasks applicability 

Fig. 3: Hierarchical taxonomy ofAEC tasks and 
operations [37] 
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According to the authors, the level of composite tasks is 
appropriate to assess the suitability of mixed reality 
technology and taxonomy evidences the needs of the 
practice required in the context, at different levels of 
complexity, by fostering the means to study them 
individually, and relating them to technology alternatives. 

4.1. The proposed taxonomy 

The proposed hierarchical taxonomy classifies the 
required work in the field of crack surveillance at five 
levels: Domain, Process, Activity, Sub-activities and 
Tasks, as shown in Fig. 4. It represents all the operational 
work required in the domain, which is composed of 
processes made through activities that are divided into 
sub activities; subactivities are composed of tasks 
performed to meet the needs of the domain. 

Cracks surveillance is carried out through processes 
already consolidated in the context of dam safety, 
required by norms and laws that regulate and supervise 
the practices. The subset of the processes for surveillance 
of concrete dams that includes, among other things, the 
suveillance of cracks are: Mapping , Periodic inspection 
and Monitoring , as highlighted in Fig. 4. 

Mapping is handled by the activities C rack detection, 
Crack Characterization, Crack survey, Validate and 
Analyze behaviour. On the other hand, the periodic 


inspection comprises the activities Analyze Document, 
Anomaly detection, Crack Characterization, Crack 
survey, Analyze behavior and Validate. Lastly, 
Monitoring occurs by means of activities: Read 
instrument, Validate and Analyze behaviour. 

The proposed hierarchical taxonomy also has an 
intermediate level between the level of activities and the 
level of tasks, called Sub-activities. The sub-activities 
required in the activities are: Surface inspection, Crack 
registration, Acquire location, Locate, Acquire openess, 
acquire length, acquire depth, Direction registration, 
Acquire image, Appearance registration, Age estimation, 
Cause estimation, Data examination, Data processing, 
Document examination, Advise, Send review, Develop 
report, Acquire reading, Routine inspection, Register 
event, Repeat reading, Check reading. 

At the last level of the taxonomy are the tasks to meet 
sub-activities: Draw, Move to, Observe, Recognize, 
Photograph, Detect, Measure, Indicate, Annotate, Search, 
Research, Select, Prepare, Inform, Compare. 

The relationship between the elements of the taxonomy 
can be seen in Fig. 4. The links represented by dashed 
lines represent works that depend on specific facts to be 
performed. 


Sub-activities 



- Compare 


Fig. 4: Hierarchical taxonomy of tasks of the surveillance of cracks in concrete dams 


4.2. Classifying tasks 

Task is the smallest unit of work considered in the 
proposed taxonomy. It represents aportion of the physical 
work (interactions with the environment) and mental 


(internal interactions involving the perceptual and 
mediational system) that the user needs to perform in a 
sub-activity. 
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The way in which the work happens depends on several 
extrinsic factors, such as characteristics of the 
environment where the task is performed and resources 
available to help the user execute them The way the work 
happens depends also on factors intrinsic to the user, such 
as skills, experience and knowledge, among others. The 
tasks of the domain were classified according to the units 
of behavior of the taxonomy of Berliner et al., as shown 
in Table 1. 


Table 1: Classification of the tasks of cracks surveillance 
in concrete dams on Berliner's taxonomy 
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V. CONCLUSION 

There are several studies demonstrating the potential for 
augmented reality in the AEC / FM area. Existing 
solutions already explore augmented reality functions to 
optimize the activities in which they are applied. 
However, there is still room for research that seeks to 
broaden the understanding of both augmented reality 
technology and the domain to which these interfaces will 
be applied. The proposed hierarchical taxonomy and 
classification of tasks regarding Berliner taxonomy 
provides resources to identify the tasks with the greatest 
impact by using the technology. Since augmented reality 
is a technology that allows information access and 
visualization, it is concluded that structuring the 
knowledge about the perceptive, cognitive and motor 
workload of the tasks of the domain, allows the 
application of technology in a more conscious way, 
increasing the contributions provided by the augmented 
reality, favoring the successful application of the 
technology in the surveillance of cracks in concrete dams. 
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